feat(data-warehouse): implement electricity_maps import source by Gilbert09 · Pull Request #107247 · PostHog/posthog

Gilbert09 · 2026-09-26T10:45:13Z

Problem

Users can't sync grid carbon data into the data warehouse: the Electricity Maps connector was a hidden scaffold (unreleasedSource=True) with no fields and no sync logic.

Changes

Users can now connect Electricity Maps in the sources wizard and sync two tables, carbon_intensity and power_breakdown, one row per hour per configured grid zone. The source ships visible with releaseStatus=ALPHA; the unreleasedSource flag is removed.
The connection form takes an API token, a comma-separated zone list, and an optional initial history depth (default 30 days). Zone access and history depth are gated by the vendor plan, so validation probes each zone with the cheap /carbon-intensity/latest call and maps 401/403/404 to actionable messages; the same statuses are non-retryable at sync time.
The transport is the shared rest_source framework with a custom paginator: the vendor's past-range endpoints have no pagination and cap one request at 10 days, so the paginator walks 5-day windows forward.
The walk is window-major (every zone per window, then the next window). The pipeline checkpoints the ascending incremental watermark per batch, so a zone-major walk could strand later zones a full backfill behind the watermark after a crash. Window-major bounds that loss to one window.
Incremental sync uses the server-side start/end datetime filter, rows merge on (zone, datetime), tables partition by month on datetime, and interrupted syncs resume from a saved (window, zone) position.
Mechanical: regenerated generated_configs/electricitymaps.py, canonical table/column descriptions from the vendor docs, and the SOURCES.md row moved from Scaffolded to Implemented. hogli build:openapi produced no diff because the source type was already in the generated enum.

Note

Endpoint behavior (params, response shape, the 10-day range cap) was verified against the vendor's API reference and unauthenticated probes of the live host. Full round-trips with a real token were not verified because no credentials were available, which is part of why the source ships as alpha.

How did you test this code?

New transport tests (tests/test_electricity_maps.py) catch: a paginator ordering change that would corrupt the incremental watermark (window-major walk, final-window clamp, one valid window when the watermark reaches now), resume state not saved or not honored, an incremental sync that ignores the watermark or a full refresh that uses it, and wrong user-facing messages per credential-probe status. No existing test covers this source.
New source tests (tests/test_electricity_maps_source.py) cover the credential-validation branches that fail before any API call (empty and malformed zone lists).
The registry-wide invariant suite (sources/tests/, covering categories, version metadata, generated configs, and source loading) passes with the new source.
python manage.py audit_source_docs against a posthog.com checkout reports no finding for this source; the remaining findings are pre-existing drift for other sources.
Not run: a live sync against the Electricity Maps API, because the environment has no credentials.

Release status

No feature flag controls this change
This change is behind a feature flag and is not available to users
This change makes a previously flagged feature available to everyone

Automatic notifications

Publish to changelog?

Docs update

The user-facing doc is PostHog/posthog.com#20493, matching this source's docsUrl of /docs/cdp/sources/electricity-maps.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Claude Fable 5 (claude-fable-5)

Skills invoked: implementing-warehouse-sources, documenting-warehouse-sources, /writing-tests, /writing-user-facing-copy, /writing-code-comments, /writing-dataclasses, /writing-pr-descriptions.
Duplicate search: gh pr list --state open --search "electricity maps" found no open PR for this source.
The vendor's docs site blocks non-browser clients, so endpoint details came from an archived copy of the official API reference plus unauthenticated probes of the live API host.
All sample data in tests is invented from the public API documentation.
CodeRabbit local pass skipped: cloud task run.

Created with PostHog Desktop

Implements the Electricity Maps warehouse source on the shared rest_source framework: hourly carbon_intensity and power_breakdown tables per configured zone, resumable window-walking past-range requests, incremental sync on datetime, canonical descriptions, tests, and SOURCES.md entry. Ships released with releaseStatus alpha. Generated-By: PostHog Desktop Task-Id: 44d1cd84-3c0e-4a53-85c7-03a29e70e82f

trunk-io · 2026-09-26T10:45:17Z

Merging to master in this repository is managed by Trunk.

To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

coderabbitai · 2026-09-26T10:45:22Z

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Warning

Review limit reached

Next included review available in 13 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 12 included reviews currently available.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: PostHog/posthog/.coderabbit.yaml

Review profile: QUIET

Plan: Enterprise

Run ID: 4f33106b-9be0-48a4-a885-a9232b4de8a5

📥 Commits

Reviewing files that changed from the base of the PR and between 076575a and 3184e48.

📒 Files selected for processing (8)

products/warehouse_sources/backend/temporal/data_imports/sources/SOURCES.md
products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/canonical_descriptions.py
products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/electricity_maps.py
products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/settings.py
products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/source.py
products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/tests/test_electricity_maps.py
products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/tests/test_electricity_maps_source.py
products/warehouse_sources/backend/temporal/data_imports/sources/generated_configs/electricitymaps.py

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: PostHog/posthog/.coderabbit.yaml

Review profile: QUIET

Plan: Enterprise

Run ID: 612c3b5c-ae6b-4b4d-8f8f-b63f62b699a4

📥 Commits

Reviewing files that changed from the base of the PR and between f14ac05 and 076575a.

📒 Files selected for processing (3)

products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/electricity_maps.py
products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/settings.py
products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/source.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 2 remain after this review.

📝 Walkthrough

Walkthrough

The pull request adds Electricity Maps as an implemented HTTP source. It supports carbon-intensity and power-breakdown endpoints, zone validation, configurable history depth, and resumable date-window pagination. Incremental runs use merge/upsert writes, while other runs use replacement writes. The source includes API credential checks, endpoint descriptions, configuration fields, and tests for pagination, resume state, zone handling, and validation.

Priority: ⬇️ Low

Merge Risk: 🟡 Moderate · up to 07657

Changing zone order during a retry can leave historical Electricity Maps rows permanently missing; preserve zone identity in checkpoints before merging.

Security Architecture Review

Security architecture risk: 🟡 Moderate · up to 07657

Credentialed requests remain directed to the Electricity Maps API, but changing the configured zone order during a retry can silently omit hourly data. A later incremental sync may not recover the omitted rows automatically.

Retained concerns

Medium · reliability · inferred: A retry after zone reordering can apply a saved numeric index to a different zone, skip the originally pending zone for that window, and then advance the incremental watermark past its missing rows.

Security review details

Security Blast Radius

inferred — The reviewed credentialed network path is confined to the provider host; the data-integrity concern affects the configured source's warehouse rows, not an identified cross-tenant privilege boundary.

Trust Boundaries and Controls

observed — Extraction pins the fixed API host and disables redirects. Validation uses the fixed URL with redirects disabled and redacts the token in its tracked session.

Resilience and Maintainability Implications

observed — Resume state is scoped by team, job, and endpoint namespace, and saved cursors expire after 24 hours. Successful source traversal does not itself clear the endpoint cursor.

Hardening Proposals

proposed — Bind a saved position to stable zone identity or reject a cursor when the configured zone set or order changes; clear the endpoint cursor after successful completion without clearing it before the final rows are durable.

🚥 Pre-merge checks | ✅ 1

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description is complete and follows the required structure. It explains the problem, user-visible changes, testing, release status, documentation, agent context, and known testing limitation. It d…

✨ Finishing Touches

📝 Generate docstrings

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-09-26T10:45:26Z

🤖 CI report

⚠️

Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

⚠️

Duplication (Python) — 2 new duplicated blocks (worst 162 tokens)

New Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

First copy	Second copy	Lines	Tokens
`products/warehouse_sources/backend/temporal/data_imports/sources/companycam/source.py:1`	`products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/source.py:1`	21	162
`products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/electricity_maps.py:9`	`products/warehouse_sources/backend/temporal/data_imports/sources/servicem8/servicem8.py:8`	12	138

✅ Duplication (TypeScript) — clean

New TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

⚠️

Comment density — 4% of added code lines are comments (29 of 736)

This section warns when comments are more than 3% of the code lines a PR adds, and alerts above 6%. Before agent-assisted PRs, the typical share was about 2%. Only full-line comments count. Docstrings, generated files, snapshots, migrations, and workflow files are left out.

Comments that restate the code, record how the change came about, or narrate the next line add noise for the next reader. Keep the comments that explain a reason the code cannot show, and remove the rest. See .agents/skills/writing-code-comments/SKILL.md for the house rules.

Files with the most added comment lines:

File	Comment lines	Added lines
`products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/electricity_maps.py`	15	239
`products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/settings.py`	7	29
`products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/tests/test_electricity_maps.py`	5	246
`products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/source.py`	2	133

This check does not block merging. It updates on every push and clears when the share drops.

⚠️

Backend coverage — 98.0% of changed backend lines covered — 5 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ████████████████████ 98.0% (304 / 309)

File	Patch	Uncovered changed lines
`products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/source.py`	93.9%	98, 108
`products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/electricity_maps.py`	97.6%	106, 262–263

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 36318196347 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
`demo`	`████████████░░░░░░░░` 57.8%	1,545 / 2,673
`batch_exports`	`████████████████░░░░` 81.2%	21,453 / 26,424
`cdp`	`██████████████████░░` 88.2%	4,548 / 5,155
`mcp_analytics`	`██████████████████░░` 88.9%	4,910 / 5,523
`product_tours`	`██████████████████░░` 89.3%	1,331 / 1,491
`dashboards`	`██████████████████░░` 89.5%	6,839 / 7,641
`signals`	`██████████████████░░` 89.8%	54,454 / 60,606
`data_warehouse`	`██████████████████░░` 89.9%	13,912 / 15,470
`notebooks`	`██████████████████░░` 90.2%	15,287 / 16,945
`cohorts`	`██████████████████░░` 90.4%	8,420 / 9,316
`streamlit_apps`	`██████████████████░░` 90.7%	2,625 / 2,895
`managed_warehouse`	`██████████████████░░` 90.9%	10,215 / 11,234
`tasks`	`██████████████████░░` 91.1%	73,947 / 81,146
`data_modeling`	`██████████████████░░` 91.5%	10,525 / 11,498
`business_knowledge`	`██████████████████░░` 91.6%	6,899 / 7,528
`engineering_analytics`	`██████████████████░░` 91.7%	11,002 / 11,999
`exports`	`██████████████████░░` 91.8%	9,685 / 10,555
`conversations`	`███████████████████░` 92.5%	28,726 / 31,047
`early_access_features`	`███████████████████░` 92.6%	1,341 / 1,448
`canvas`	`███████████████████░` 92.8%	6,877 / 7,409
`approvals`	`███████████████████░` 93.0%	3,919 / 4,214
`mcp_registry`	`███████████████████░` 93.1%	1,670 / 1,794
`error_tracking`	`███████████████████░` 93.1%	15,783 / 16,950
`notifications`	`███████████████████░` 93.2%	1,145 / 1,229
`slack_app`	`███████████████████░` 93.2%	13,677 / 14,674
`stamphog`	`███████████████████░` 93.2%	7,885 / 8,456
`surveys`	`███████████████████░` 93.3%	6,571 / 7,040
`context_layer`	`███████████████████░` 93.8%	3,373 / 3,595
`web_analytics`	`███████████████████░` 93.9%	21,653 / 23,051
`alerts`	`███████████████████░` 94.0%	8,541 / 9,082
`billing_alerts`	`███████████████████░` 94.1%	2,094 / 2,226
`mcp_store`	`███████████████████░` 94.4%	8,940 / 9,472
`ai_observability`	`███████████████████░` 94.4%	22,139 / 23,454
`wizard`	`███████████████████░` 94.7%	6,151 / 6,496
`reminders`	`███████████████████░` 94.8%	760 / 802
`workflows`	`███████████████████░` 94.8%	13,615 / 14,360
`review_hog`	`███████████████████░` 94.9%	11,490 / 12,109
`annotations`	`███████████████████░` 95.1%	817 / 859
`customer_analytics`	`███████████████████░` 95.1%	24,756 / 26,028
`endpoints`	`███████████████████░` 95.1%	9,211 / 9,681
`marketing_analytics`	`███████████████████░` 95.3%	19,216 / 20,161
`posthog_ai`	`███████████████████░` 95.4%	2,489 / 2,610
`tracing`	`███████████████████░` 95.4%	3,483 / 3,650
`growth`	`███████████████████░` 95.4%	9,812 / 10,282
`logs`	`███████████████████░` 95.4%	15,290 / 16,022
`experiments`	`███████████████████░` 95.5%	32,956 / 34,526
`actions`	`███████████████████░` 95.5%	756 / 792
`data_catalog`	`███████████████████░` 95.6%	4,402 / 4,606
`skills`	`███████████████████░` 95.8%	6,972 / 7,274
`replay_vision`	`███████████████████░` 96.0%	26,757 / 27,885
`product_analytics`	`███████████████████░` 96.2%	28,495 / 29,617
`autoresearch`	`███████████████████░` 96.2%	8,115 / 8,434
`revenue_analytics`	`███████████████████░` 96.4%	1,876 / 1,946
`access_control`	`███████████████████░` 96.4%	7,122 / 7,386
`user_interviews`	`███████████████████░` 96.5%	2,859 / 2,963
`feature_flags`	`███████████████████░` 96.5%	25,499 / 26,416
`warehouse_sources`	`███████████████████░` 97.2%	448,161 / 460,961
`data_quality`	`████████████████████` 97.7%	7,592 / 7,774
`metrics`	`████████████████████` 98.1%	4,085 / 4,166
`analytics_platform`	`████████████████████` 98.3%	2,783 / 2,832
`pulse`	`████████████████████` 98.5%	2,043 / 2,075
`live_debugger`	`████████████████████` 99.2%	626 / 631

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

veria-ai · 2026-09-26T10:51:51Z

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

trunk-io · 2026-09-26T11:02:10Z

_{View Full Report ↗︎ ⋅ Docs}

coderabbitai

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments (1)

products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/electricity_maps.py-113-140 (1)

113-140: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Invalidate the resume checkpoint when the Electricity Maps zone list changes.

A source update can change job_inputs["zones"]. A retry reuses the same ExternalDataJob and resumable key, while the import reads the current model.pipeline.job_inputs. set_resume_state() checks only whether the saved positional index is in range. For example, inserting a zone before the saved position keeps the index valid but skips the inserted zone for that window. Persist a zone-list fingerprint with the checkpoint and restart the window when it differs.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: PostHog/posthog/.coderabbit.yaml

Review profile: QUIET

Plan: Enterprise

Run ID: cb48e359-33ac-4e5a-944b-5a4c6effc044

📥 Commits

Reviewing files that changed from the base of the PR and between fc46b49 and e814682.

📒 Files selected for processing (8)

products/warehouse_sources/backend/temporal/data_imports/sources/SOURCES.md
products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/canonical_descriptions.py
products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/electricity_maps.py
products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/settings.py
products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/source.py
products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/tests/test_electricity_maps.py
products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/tests/test_electricity_maps_source.py
products/warehouse_sources/backend/temporal/data_imports/sources/generated_configs/electricitymaps.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Matches the extraction client's redirect boundary: `validate_credentials` sends the same auth-token header as the sync client but was using the session default (`allow_redirects=True`), so a redirect from the vendor host could carry the token to another origin. Pin `allow_redirects=False`, same as `electricity_maps_source`'s REST client config. Addresses a CodeRabbit finding on PR #107247.

A new stamphog review started for this PR — the fresh verdict replaces this approval.

Matches the extraction client's redirect boundary: `validate_credentials` sends the same auth-token header as the sync client but was using the session default (`allow_redirects=True`), so a redirect from the vendor host could carry the token to another origin. Pin `allow_redirects=False`, same as `electricity_maps_source`'s REST client config. Addresses a CodeRabbit finding on PR #107247. Fixes a ruff formatting failure from the previous push.

A new stamphog review started for this PR — the fresh verdict replaces this approval.

Gilbert09 · 2026-09-27T11:29:51Z

Summary of follow-up on the automated reviews:

veria-ai's "unbounded request fan-out" (resolved inline thread): declined — no other source in the registry caps history-depth (rki_covid, metronome), the real-world zone list is small and fixed, and the "monopolize a worker" concern is already covered by the platform-wide Temporal heartbeat/start-to-close timeouts, not something to special-case per connector. Full reasoning is on the thread.
CodeRabbit's architecture review, "credential validation follows the session's default redirect behavior": fixed in 5c139ee/401cf6e — validate_credentials now passes allow_redirects=False to make_tracked_session, matching the extraction client's redirect boundary for the same auth-token header.
CodeRabbit's architecture review, "resume cursor is positional rather than bound to zone identity": not changing for this PR. The current design is intentional and tested (test_resume_with_out_of_range_zone_index_falls_back_to_first_zone): a zone list edit between save and resume is expected to happen rarely, an out-of-range index already falls back safely, and the residual risk CodeRabbit flags (an in-range reorder, not a removal) is bounded to one window of data (5 days) for the reordered zone, self-heals on the next incremental sync, and recoverable at any time with a full refresh. Re-keying resume state to zone identity is a reasonable future hardening but is a larger, separate change or this PR, not a defect it introduces.

CI is green and the branch is up to date with master.

_{🦉 via talyn.dev}

A new stamphog review started for this PR — the fresh verdict replaces this approval.

coderabbitai

Actionable comments posted: 3

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: PostHog/posthog/.coderabbit.yaml

Review profile: QUIET

Plan: Enterprise

Run ID: 1addbca7-ba9e-4816-9dcf-613e9b754ef6

📥 Commits

Reviewing files that changed from the base of the PR and between e814682 and f14ac05.

📒 Files selected for processing (2)

products/warehouse_sources/backend/temporal/data_imports/sources/SOURCES.md
products/warehouse_sources/backend/temporal/data_imports/sources/electricity_maps/electricity_maps.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 7 remain after this review.

…st timeouts Addresses two CodeRabbit findings on PR #107247: - `get_resumable_source_manager` now namespaces by `inputs.schema_name`, matching the established pattern for other multi-endpoint resumable sources (see `beehiiv`, `zoho_crm`, etc.). Without it, `carbon_intensity` and `power_breakdown` shared one Redis resume key, so a retry could load the other endpoint's `(window_start, zone_index)` cursor and skip or replay the wrong rows. - Both the extraction client and the credential-validation probe now set a bounded `(connect, read)` timeout (`REQUEST_TIMEOUT_SECONDS = (10.0, 60.0)`), matching the convention used by nearly every other REST source in the registry. Previously neither request had a timeout, so a stalled vendor response could hold the import worker (or a connection check) indefinitely.

A new stamphog review started for this PR — the fresh verdict replaces this approval.

stamphog

Approved.

Self-contained new data-warehouse connector (own directory, no shared framework changes) with solid safety practices (redirect pinning, token redaction, per-zone error mapping) and thorough new tests; author is on the owning team with STRONG familiarity (88% of touched lines, 98 recent merged PRs in this area), and both CodeRabbit and a security bot's final passes show no outstanding actionable concerns, with the one declined suggestion (resume checkpoint edge case) reasonably justified and covered by an existing test.

Author wrote 88% of the modified lines and has 98 merged PRs in these paths (familiarity STRONG).

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✓	no deny categories matched
size	✓	524L, 5F substantive, 866L/8F incl. docs/generated/snapshots — within ceiling
tier	✓	T1-agent / T1d-complex (866L, 8F, single-area, feat)
stamphog 2.2.0		`.stamphog/policy.yml` @ `3184e48` · reviewed head `3184e48`

Copilot AI lite review requested due to automatic review settings September 26, 2026 10:45

Gilbert09 self-assigned this Sep 26, 2026

Gilbert09 added the stamphog Request AI approval (no full review) label Sep 26, 2026 — with Talyn App

Copilot AI reviewed Sep 26, 2026

Gilbert09 mentioned this pull request Sep 26, 2026

docs(cdp): add Electricity Maps source doc PostHog/posthog.com#20493

Draft

5 tasks

pr-assigner-resolver-posthog Bot requested a review from a team September 26, 2026 10:45